{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Copyright 2018-2019 IBM Corp. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Object Detection - Visualize Results\n",
    "\n",
    "This example illustrates the usage of [MAX Object Detector](https://developer.ibm.com/exchanges/models/all/max-object-detector) model. This notebook guides you through running the model on a sample image to get the objects, extracting the bounding boxes and then visualizing them over the image.\n",
    "\n",
    "\n",
    "## Setup\n",
    "\n",
    "The notebook calls the `MAX Object Detector` microservice, which must be running. You can either use the [hosted demo instance](http://max-object-detector.codait-prod-41208c73af8fca213512856c7a09db52-0000.us-east.containers.appdomain.cloud), or follow the instructions for [deploying the microservice locally from the Dockerhub image](https://github.com/IBM/MAX-Object-Detector#deploy-from-docker-hub). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This notebook requires matplotlib, Pillow and requests\n",
    "# You only need to run the line below to install these if you don't already have them installed\n",
    "\n",
    "! pip install -q matplotlib Pillow requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "from PIL import Image\n",
    "import matplotlib\n",
    "from matplotlib import pyplot as plt\n",
    "import matplotlib.patches as patches\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This url must point to a running instance of the model microservice\n",
    "# By default this assumes you are using the hosted demo instance\n",
    "# If you want to use the model that is running locally, pass the `local_port` field.\n",
    "\n",
    "\n",
    "def call_model(input_img, local_port=None):\n",
    "    \"\"\"\n",
    "    Takes in input image file path, posts the image to the model and returns face bboxes and emotion predictions\n",
    "    If local port is not specified, uses the long running instance.\n",
    "    If local port is specified, uses the local instance.\n",
    "    \"\"\"\n",
    "    if local_port:\n",
    "        url = 'http://localhost:'+ str(local_port)+'/model/predict'\n",
    "    else:\n",
    "        url = 'http://max-object-detector.codait-prod-41208c73af8fca213512856c7a09db52-0000.us-east.containers.appdomain.cloud/model/predict'\n",
    "\n",
    "    files = {'image': ('image.jpg', open(input_img, 'rb'), 'images/jpeg') }\n",
    "    r = requests.post(url, files=files).json()\n",
    "    \n",
    "    return r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Visualizing the test image\n",
    "\n",
    "First we load the image with Pillow and display the image in our notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_path = 'samples/baby-bear.jpg'\n",
    "image = Image.open(img_path)\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Call model to detect objects in the image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_response = call_model(img_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Visualize model response\n",
    "\n",
    "The model returns JSON containing a `predictions` field which is an array of JSON objects, one for each object detected in the image. For each object, the bounding box coordinates are contained in the `detection_box` field, while the object labels can be found in the `label` and `label_id` fields.\n",
    "\n",
    "The bounding box coordinates are given in the format `[ymin, xmin, ymax, xmax]`, where each coordinate is _normalized_ by the appropriate image dimension (height for `y` or width for `x`). Each coordinate is therefore in the range `[0, 1]`. In order to use these coordinates to display the bounding boxes, we must first map them back to the same range as the original image, so that they become pixel coordinates.\n",
    "\n",
    "Additionally there is also a `probability` field which indicates the model's confidence of the particular object's class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Explore the model results - there should be 2 entries in the `predictions` array\n",
    "import json\n",
    "print(json.dumps(model_response, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We display bounding boxes and the class label with the predicted probability for each object.\n",
    "\n",
    "# Get the image height and width\n",
    "image_width, image_height = image.size\n",
    "# Create figure and axes\n",
    "fig, ax = plt.subplots()\n",
    "# Set larger figure size\n",
    "fig.set_dpi(600)\n",
    "# Display the image\n",
    "plt.imshow(image)\n",
    "\n",
    "# Set up the color of the bounding boxes and text\n",
    "color = '#00FF00'\n",
    "# For each object, draw the bounding box and predicted class together with the probability\n",
    "for prediction in model_response['predictions']:\n",
    "    bbox = prediction['detection_box']\n",
    "    # Unpack the coordinate values\n",
    "    y1, x1, y2, x2 = bbox\n",
    "    # Map the normalized coordinates to pixel values: scale by image height for 'y' and image width for 'x'\n",
    "    y1 *= image_height\n",
    "    y2 *= image_height\n",
    "    x1 *= image_width\n",
    "    x2 *= image_width\n",
    "    # Format the class probability for display\n",
    "    probability = '{0:.4f}'.format(prediction['probability'])\n",
    "    # Format the class label for display\n",
    "    label = '{}'.format(prediction['label'])\n",
    "    label = label.capitalize()\n",
    "    # Create the bounding box rectangle - we need the base point (x, y) and the width and height of the rectangle\n",
    "    rectangle = patches.Rectangle((x1, y1), x2 - x1, y2 - y1, linewidth=1, edgecolor=color, facecolor='none')\n",
    "    ax.add_patch(rectangle)\n",
    "    # Plot the bounding boxes and class labels with confidence scores\n",
    "    plt.text(x1, y1-5, label, fontsize=4, color=color, fontweight='bold',horizontalalignment='left')\n",
    "    plt.text(x2, y1-5, probability, fontsize=4, color=color, fontweight='bold',horizontalalignment='right')\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
